Device, method, and program for generating multidimensional reaction-type image, and method, and program for reproducing multidimensional reaction-type image

ABSTRACT

The present disclosure related to a method for playing a multidimensional reaction-type image. The method includes at least: receiving, by a computer, input manipulation to an object from a user; and extracting, by the computer, an image frame matched to a detailed cell corresponding to location information and depth information in a reaction-type image, depending on the location information and the depth information of the input manipulation received at each playback time point. The depth information is information about pressure strength of the input manipulation applied to the reaction-type image or time length to which the input manipulation is applied. The location information is information about a location of a two-dimensional space in which the input manipulation is applied to the reaction-type image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional application of U.S. patent applicationSer. No. 16/331,475 filed on Mar. 7, 2019, which is the National Phaseof International Application No. PCT/KR2017/009837, filed Sep. 7, 2017,which includes a claim of priority under 35 U.S.C. § 119(a) and § 365(b)to Korean Patent Application Nos. 10-2016-0115235, filed Sep. 7, 2016,and 10-2017-0114671, filed Sep. 7, 2017, in the Korean IntellectualProperty Office. The above-listed applications are incorporated hereinby reference in their entirety.

BACKGROUND

Embodiments of the inventive concept described herein relate to adevice, a method and a program for generating a multidimensionalreaction-type image, and a method and a program for reproducing amultidimensional reaction-type image.

Recently, a technology for capturing an image has been developeddramatically. Not only camcorders and digital cameras, but also mobileterminals such as smart phones may also capture high resolution images.Furthermore, a 360-degree camera, a 3D image camera and the like areemerging.

The image is captured by an image capturing device and stored in aspecific format, and played by a playable terminal. Image playback isprovided unilaterally in time order without interaction with viewers. Inother words, the viewers may sense only the visual feeling throughplaying images.

SUMMARY

Embodiments of the inventive concept provide a device, a method and aprogram for generating a multidimensional reaction-type image, and amethod and a program for reproducing a multidimensional reaction-typeimage that generate an image respond to input manipulation of a user asan image frame corresponding to manipulation to a specific region of anobject is connected, and then provide the user.

According to an exemplary embodiment, a method for generating amultidimensional reaction-type image includes obtaining, by a computer,a plurality of multidimensional image frames forming a base image,wherein the base image is an image from applying manipulation to anobject through a manipulation object, generating, by the computer, athree-dimensional cell combination based on a configuration regionwithin the base image and a specific frame density within theconfiguration region, wherein the three-dimensional cell combinationincludes a plurality of detailed cells to which different depthinformation and different location information are assigned, andmatching, by the computer, a respective image frame included in the baseimage to a corresponding detailed cell. The depth information isinformation about pressure strength of input manipulation applied to areaction-type image or time length to which the input manipulation isapplied. The location information is information about a location of atwo-dimensional space in which the input manipulation is applied to thereaction-type image. The configuration region is a two-dimensional spacearea generated as the reaction-type image in an object, and the framedensity is the number of steps of the depth information applied to aspecific point in the base image.

In another embodiment, the multidimensional image frame is repeatedlyobtained while a location and the pressure strength, at whichmanipulation is applied to a specific object through a manipulationobject, are changed.

In another embodiment, the obtaining of the multidimensional imageframes includes extracting a restoration image, which is restored afterthe input manipulation is applied in an entire image in which the inputmanipulation is applied to the object through the manipulation objectand obtaining a plurality of image frames in the restoration image.

In another embodiment, the method further includes assigning theconfiguration region to be generated as the reaction-type image.

In another embodiment, the method further includes calculating arestoration variable of the object by recognizing a change within thebase image after pressure of specific strength is applied.

According to an exemplary embodiment, a multidimensional reaction-typeimage generating program is coupled to hardware and is stored in mediato perform the above-described multidimensional reaction-type imagegenerating method.

According to an exemplary embodiment, a method for playing amultidimensional reaction-type image includes receiving, by a computer,input manipulation to an object from a user and extracting, by thecomputer, an image frame matched to a detailed cell corresponding tolocation information and depth information in a reaction-type image,depending on the location information and the depth information of theinput manipulation received at each playback time point. The depthinformation is information about pressure strength of the inputmanipulation applied to the reaction-type image or time length to whichthe input manipulation is applied. The location information isinformation about a location of a two-dimensional space in which theinput manipulation is applied to the reaction-type image. A specificimage frame corresponding to the location information and the depthinformation of the input manipulation is matched to the detailed cell,and the detailed cell constitutes a three-dimensional cell combinationin the reaction-type image.

In another embodiment, the method further includes generating, by thecomputer, a final playback image frame by performing morphing based on aplurality of image frame combinations, the depth informationcorresponding to a specific point on a screen of each of which isdifferent, or a combination of an image frame provided at a previoustime point and an image frame within the detailed cell corresponding toa current input manipulation, when successive input manipulations areentered into an adjacent region.

In another embodiment, the method further includes adjusting, by thecomputer, speed of image frame change by extracting a restorationvariable from the reaction-type image.

In another embodiment, the method further includes adjusting, by thecomputer, an image frame change according to the input manipulation tothe object by calculating time elapsing from a specific time point atwhich the reaction-type image is executed.

According to an exemplary embodiment, a reaction-type image generatingprogram is coupled to hardware and is stored in media to perform theabove-described multidimensional reaction-type image playing method.

BRIEF DESCRIPTION OF THE FIGURES

The above and other objects and features will become apparent from thefollowing description with reference to the following figures, whereinlike reference numerals refer to like parts throughout the variousfigures unless otherwise specified, and wherein:

FIG. 1 is a flowchart of a method for generating a multidimensionalreaction-type image, according to an embodiment of the inventiveconcept;

FIG. 2 is an exemplary view for generating a plurality of segmentedimage to divide a reaction-type generation region into a plurality ofplayback regions, according to an embodiment of the inventive concept;

FIG. 3 is a flowchart of a method for generating a multidimensionalreaction-type image, which further includes a procedure of assigning areaction-type generation region, according to an embodiment of theinventive concept;

FIG. 4 is a flowchart of a method for generating a multidimensionalreaction-type image, which further includes a procedure of calculating arestoration variable, according to an embodiment of the inventiveconcept;

FIG. 5 is a flowchart of a method for playing a multidimensionalreaction-type image, according to an embodiment of the inventiveconcept;

FIG. 6 is an exemplary view illustrating a procedure of determining adetailed cell for extracting an image frame based on locationinformation and depth information of input manipulation, according to anembodiment of the inventive concept;

FIG. 7 is a flowchart of a multidimensional reaction-type image playingmethod, which further includes an image frame morphing procedure,according to an embodiment of the inventive concept;

FIG. 8 is a flowchart of a method for playing a multidimensionalreaction-type image, which further includes a procedure of adjustingimage frame playback through a restoration variable, according to anembodiment of the inventive concept; and

FIG. 9 is a flowchart of a method for playing a multidimensionalreaction-type image, which further includes a procedure of adjusting animage frame change according to input manipulation by reflecting anobject state change with time, according to an embodiment of theinventive concept.

DETAILED DESCRIPTION

Hereinafter, exemplary embodiments of the inventive concept will bedescribed in detail with reference to the accompanying drawings. Theabove and other aspects, features and advantages of the inventiveconcept will become apparent from the following description of thefollowing embodiments given in conjunction with the accompanyingdrawings. However, the inventive concept is not limited to theembodiments disclosed below, but may be implemented in various forms.The embodiments of the inventive concept is provided to make thedisclosure of the inventive concept complete and fully inform thoseskilled in the art to which the inventive concept pertains of the scopeof the inventive concept. The same reference numerals denote the sameelements throughout the specification.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by thoseskilled in the art to which the inventive concept pertains. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the specification andrelevant art and should not be interpreted in an idealized or overlyformal sense unless expressly so defined herein.

The terms used herein are provided to describe the embodiments but notto limit the inventive concept. As used herein, the singular terms areintended to include the plural forms as well, unless the context clearlyindicates otherwise. The terms “comprises” and/or “comprising” usedherein does not exclude presence or addition of one or more otherelements, in addition to the aforementioned elements.

In this specification, a ‘computer’ includes all the various devicescapable of performing arithmetic processing. For example, the computermay correspond to not only a desktop personal computer (PC) or anotebook but also a smart phone, a tablet PC, a cellular phone, apersonal communication service phone (PCS phone), a mobile terminal of asynchronous/asynchronous International Mobile Telecommunication2000(IMT-2000), a palm PC, a personal digital assistant (PDA), and the like.Furthermore, a computer may correspond to a server computer thatreceives information from a client. Hereinafter, in this specification,a computer may be represented as a terminal or client.

In this specification, a ‘reaction-type image’ refers to an image playedin a form corresponding to a specific input manipulation of a user(i.e., a viewer) who watches an image. For example, the reaction-typeimage refers to an image in which a motion of touching an object isplayed, such as an input manipulation to a touch screen, when inputmanipulation of touching a specific object (e.g., pillow) is applied tothe touch screen by the user. Moreover, for example, the reaction-typeimage refers to an image in which motion of pressing an object andmotion of restoring the object after the input manipulation of the userare played, when input manipulation of pressing a specific object isapplied to the touch screen by the user.

In this specification, the ‘base image’ refers to a plurality of imageframes combinations used to generate reaction-type images.

In this specification, the ‘first client’ refers to the client device ofthe first user (i.e., the reaction-type image creator) that generatesthe reaction-type image.

In this specification, the ‘second client’ refers to the client deviceof the second user (i.e., the reaction-type image user) that plays thereaction-type image.

In this specification, the ‘input manipulation’ refers to themanipulation to the image of the user received through the input meansof the computer playing the reaction-type image. For example, the inputmanipulation may include manipulation (e.g., click manipulation, dragmanipulation, contact touch manipulation, force touch manipulation(i.e., touch manipulation that applies a specific pressure to a touchscreen or touch pad)) that can be entered into a specific point orregion in the image, via the input means such as a mouse or touchscreen. Moreover, for example, the input manipulation may include thearrangement state or motion of a terminal itself, which can be obtainedusing a sensor (e.g., an acceleration sensor, a gyro sensor, or thelike) provided in a computer (or a terminal).

In this specification, the ‘object’ means an object in the reaction-typeimage, which is manipulated by the user. For example, in the case of animage from capturing an operation in which the user's hand touches aspecific object, the object refers to an object touched by the user.

In this specification, the ‘manipulation object’ is for performingmanipulation or motion on an object in an image. For example, whentouching or pressing a bag or pillow in an image by using a hand, themanipulation object means the hand touching the bag or pillow.

Hereinafter, according to an embodiment of the inventive concept, adevice, a method and a program for generating a multidimensionalreaction-type image, and a method and a program for reproducing amultidimensional reaction-type image will be described with reference todrawings.

FIG. 1 is a flowchart of a method for generating a multidimensionalreaction-type image, according to an embodiment of the inventiveconcept.

Referring to FIG. 1, according to an embodiment of the inventiveconcept, a method for generating a multidimensional reaction-type imageincludes acquiring, by a computer, a plurality of multidimensional imageframes forming a base image (S200), generating a three-dimensional cellcombination on the basis of a configuration region in the base image anda specific frame density in the configuration region (S600), andmatching image frames corresponding to respective detailed cells(S1000). Hereinafter, the detailed description of each operation isprovided.

In operation S200, the computer acquires a plurality of multidimensionalimage frames forming a base image. That is, the computer may obtain amultidimensional image frame via a camera, and may extract a pluralityof image frames from an already acquired single image. For example, thecomputer may capture an image from manipulating a specific objectthrough a specific manipulation object (e.g., a hand, a stick, an airpressure supply device, or the like).

In one embodiment of the method of generating the base image, whilechanging any one of various conditions, the computer obtains an image,in which specific manipulation (e.g., touching manipulation or pressingmanipulation) is applied to a specific object through using amanipulation object, several times. That is, the computer may capture animage while changing only one condition of two-dimensional spatialmotion (i.e., X-axis direction motion and Y-axis direction motion) andZ-axis direction motion (i.e., pressure to press an object) in a statewhere the locations of the object and the camera are fixed.

For example, in a state where the pressure and the location on thespecific first axis (e.g., y-axis) are fixed, the computer obtains abase image moving in one direction along the second axis (e.g., thex-axis) and continuously obtains the base image while changing thepressure and the location on the specific first axis in turn. Inparticular, while fixing the pressure to press an object and fixing thevalue on the y-axis, the computer obtains an image that moves themanipulation object along the x-axis, via a camera; when movement to thex-axis direction is completed in a state where the pressure to theobject and the location on the y-axis are fixed, the computer changesthe location on the y-axis by the minimum unit, to obtain the base imagerepeatedly while moving again in the x-axis direction. Furthermore,after repeating the process of moving along the second axis whilechanging the location on the first axis, the computer can repeat theprocess of moving on the two-dimensional space by changing the pressureapplied to the object by the manipulation object. As such, the computermay generate a final base image for generating a reaction-type image byobtaining an image frame corresponding to the pressure intensity at eachlocation in a specific two-dimensional space.

In addition, in another embodiment of the method of generating the baseimage, upon generating and storing the base image, in a state where thecomputer fixes locations on the first axis (i.e., y-axis) and the secondaxis (e.g., pressure applied to the object), after the movement in thesecond axis (i.e., x-axis) direction is completed, the computer deletesan image in which the manipulation object returns to the initial pointof the first axis to adjust the location on the first axis (i.e., they-axis), without storing the image. That is, because an image frame inthe process of moving in the opposite direction on the second axis tochange the location of the first axis direction is an image frameunnecessary to generate a reaction-type image, the computer determinesthe motion direction of the manipulation object and deletes the imageframe moving in the direction opposite to the direction in which theimage frame is obtained, without storing the image frame. As such, afile size may be reduced by decreasing the number of image framesincluded in the reaction-type image.

Also, in another embodiment, the computer stores only the restorationimage upon generating and storing the base image. The computer storesonly the restoration image in which the state of the specific point ofthe object is restored to a state, where the external force is notapplied, after the external force is applied. In particular, thecomputer deletes the playback range to which the pressure is applied bythe manipulation object, and stores a playback range (i.e., restorationimage) that is restored after the external force by the manipulationobject is applied. The computer matches an image frame of the restoredprocess to the detailed cell for each depth of the specific point. Assuch, the computer may obtain an image frame whose surface state of thespecific point changes without being masked by the manipulation object.

Furthermore, in another embodiment, the computer may perform imageprocessing (e.g., a method of cutting and pasting a part of an objectimage corresponding to a manipulation object) using a frame in whichonly the object is present in the region masked because the manipulationobject is located, and then may generate and store an image frameincluding only the object without the manipulation object.

Moreover, in another embodiment, the computer stores the image frame foreach manipulation type so as to provide different image frames dependingon the manipulation type of the user. For example, when a user appliespressure to a specific point in a reaction-type image and then performsmanipulation to remove the pressure, the object region other than thepoint at which the pressure is applied needs not to be changed. On theother hand, when a manipulation object moves in a specific directionwhile applying pressure, an image frame in which the region where themanipulation object has passed after the pressure has already beenapplied by the manipulation object is restored needs to be provided.Accordingly, the computer stores different image frames depending oneach manipulation type (e.g., manipulation to touch a specific point,manipulation to drag a manipulation object while applying specificpressure to a touch screen) such that different image frames areprovided. To this end, the computer determines the manipulation typeentered onto the touch screen and determines the type of an image frameto be provided in turn.

Afterward, in operation S600, the computer generates a three-dimensionalcell combination within the reaction-type generation region, based onthe configuration region within the base image and the specific regiondensity within the configuration region. The configuration region is atwo-dimensional space area to which manipulation is applied by themanipulation object within an object. The frame density means the numberof frames matched at a specific point. That is, the frame densitycorresponds to the number of depths (Z-axis direction depth) given tothe configuration region or the number of steps of applied pressurestrength. In the case of an object whose shape is deformed by anexternal force, because the depth of the Z-axis direction changesdepending on the pressure strength, the frame density may correspond toeither the depth step or the pressure strength step.

The frame density may be set by the first user or may be set by thecomputer (i.e., the first client). For example, the computer may set theframe density based on the number of pressure strength steps sensed bythe force touch sensor. For example, the number of pressure strengthsteps sensed by the force touch sensor is applied at the maximum framedensity; when the number of image frames actually obtained is less thanthe frame density, the computer may allow the same image frame to bematched to a plurality of successive detailed cells.

The computer forms a three-dimensional cell combination based on thenumber of divided detailed unit and the frame density of theconfiguration region. The cell combination is divided into ‘L’ pieces bya unit interval in the x-axis direction; the cell combination is dividedinto ‘M’ pieces by a unit interval in the y-axis direction; the cellcombination is divided into ‘N’ pieces in the Z axis direction so as tocorrespond to the frame density and includes detailed cells of (L*M*N).The corresponding point and pressure strength in the configurationregion are assigned to a respective detailed cell, and the respectivedetailed cell is matched with a location suitable for the configurationregion. That is, the computer generates an array of cell combinationsand allows image frame storage locations corresponding to individualcells in the array to be connected.

Afterward, in operation S1000, the computer matches and stores the imageframe corresponding to the respective detailed cell. That is, thecomputer stores the image frame individually in each detailed cell,based on the location in the two-dimensional space of each image framein the base image, the pressure applied to the object, or Z-axisdirection depth.

Various methods of matching a frame according to pressure strength toeach detailed cell may be applied. In an embodiment, when the maximumintensity is set in the computer and the total intensity is set bydividing the total intensity into the specific number of units, thecomputer sets the pressure strength as the number of pressures isreceived from the user. Moreover, in another embodiment, the computermay allow the input manipulation to be applied during the playback ofthe base image, and may determine the location information and depthinformation corresponding to each image frame to match the detailedcell.

Moreover, in an embodiment of a method for determining a location in atwo-dimensional space, the computer recognizes the location oftwo-dimensional space where input manipulation is applied, through imagerecognition and determines that the corresponding location on the screenis the point at which the detailed cell on the screen is to be matched.

Moreover, in another embodiment, as illustrated in FIG. 2, the computergenerates a segmented image from dividing each image frame and matchesthe segmented image to each corresponding point on the touch screen, soas to divide the reaction-type generation region into the specificnumber of playback regions. At the time of image acquisition, thecomputer divides the obtained image frame into a plurality of segmentedimages by a dividing line for dividing the reaction-type generationregion and individually matches the segmented image to each point on thetouch screen. Afterward, as one or more manipulations are entered on theentire touch screen, the computer extracts and combines segmented imagescorresponding to the manipulation of each playback area to provide thecombined image on the touch screen. As such, a plurality of regions,which are divided based on one image frame source, may be implemented tobe played in multiple manners; when the input manipulation is enteredinto a plurality of points by the user, the computer may generate areaction-type image in which the reaction is generated at a plurality ofpoints.

Moreover, in another embodiment, as illustrated in FIG. 3, in operationS500, the method further includes assigning a configuration region to begenerated as a reaction-type. In an embodiment, the computer may assigna range including a region, where the appearance is changed in theobject in the base image, as a configuration region. For example, when afirst user applies an external force to an object (e.g., padding jumper)using a manipulation object (e.g., hand), the computer may determinethat only the portion (i.e., the part where the padding jumper ispressed) whose appearance changes by an external force is a region to beconverted into reaction-type by first user. Moreover, when amanipulation object is captured as well as an object in the base image,the computer may assign the configuration region by excluding the motionof the manipulation object and recognizing only the appearance change ofthe object.

Moreover, in another embodiment of assigning the configuration region,the computer extracts the object image itself that is identicallypresent in a plurality of image frames in the base image to assigns theextracted object image to the configuration region. That is, because theobject surface includes the region to be generated as a reaction-type,the computer may assign the entire range of objects included identicallyin the base image to the configuration region.

Moreover, in another embodiment of assigning the configuration region,the computer tracks the motion of the manipulation object (e.g., hand)to assign a range including the movement path of the manipulation object(e.g., hand) to the configuration region. That is, the computer extractsthe region where the manipulation object moves and then assigns theextracted region to the configuration region.

Moreover, in another embodiment of assigning the configuration region,the computer receives an input to set the reaction-type region to thespecific image frame, from the first user. For example, the computer mayextract and provide a specific image frame in the base image on thescreen and may allow a user to assign a two-dimensional range of objectsto a configuration region.

When the locations of the camera and object are fixed in the base image,the computer may provide an arbitrary image frame and then may set theregion of the object through touch manipulation. The computer mayidentically assign the configuration region assigned to a specificframe, to all the frames. Moreover, for another example, when thelocation of a camera or an object is changed, the computer may assignthe configuration region within the specific image frame, and thenautomatically assign the region within each image frame corresponding tothe image of the configuration region, to the configuration region.

Moreover, in another embodiment, as illustrated in FIG. 4, the methodfurther includes calculating (S400) a restoration variable of thecorresponding object by recognizing a change in the base image after thepressure of a specific intensity is applied. The restoration variablemeans a variable that allows the reaction-type image to be actuallyrestored after the pressure manipulation with the user's specificpressure strength on the touch screen, with respect to a motion in whichthe actual object is restored to a state where pressure is not applied.The computer (i.e., the first client) may calculate the restorationvariable to include the restoration variable in the reaction-type imageand then may provide the reaction-type image to the second client; asdescribed below, the second client may reflect the restoration variableupon playing the image frame of the reaction-type image. That is, as thesecond client plays the reaction-type image by reflecting therestoration variable, the second client may represent a reaction-typeimage to be similar to the shape from restoring an actual object'sappearance. As the pressure is applied on the touch screen by applyingthe restoration variable, the image frame is sequentially changeddepending on the pressure change, the second client plays the imageframe such that the appearance is restored to be the same to the motionof the actual object.

In an embodiment of calculating the restoration variable, the computerobtains a plurality of image frames restored after applying pressure tothe object by the manipulation object upon generating the base image,and calculates the restoration variable based on the change of theobject in the image frame per hour.

Moreover, in another embodiment of generating a restoration variable, ina process of obtaining a plurality of image frames included in the baseimage while a specific pressure is applied by the manipulation object inthe reaction-type generation region, the computer calculates therestoration variable by analyzing the object change in the region (e.g.,a region in which a hand has passed when moving with pressure applied tothe object by hand) where the manipulation object moves on the object.As such, the user does not need to perform the process of applying thepressure to the object using the manipulation object for the purpose ofcalculating the restoration variable, thereby easily generating arealistic reaction-type image.

Moreover, in another embodiment, the computer recognizes the type of thecaptured object or the motion of a manipulation object, through imagelearning and determines the restoration variable corresponding to thecharacteristic of the object. As an example in which the computer learnsan image, the computer may learn the type of an object or the motion ofa manipulation object, using a machine learning algorithm. The machinelearning algorithm includes a deep learning algorithm that performslearning based on a neural network.

For example, the computer may recognize the object included in the baseimage based on the image, which is established using big data or whichis obtained through crawling. Moreover, for example, the computeraccumulates an image of manipulation object motion (e.g., hand motion)included in a plurality of videos to perform learning and determineswhat action or manipulation the motion performed by the manipulationobject is, based on a plurality of frames corresponding to themanipulation object motion in the base image.

FIG. 5 is a flowchart of a method for playing a multidimensionalreaction-type image, according to an embodiment of the inventiveconcept.

Referring to FIG. 5, a multidimensional reaction-type image playingmethod includes receiving (S1200), by a second client, inputmanipulation to an object from a second user and sequentially providing(S1400), by the second client, image frame in the reaction-type imagebased on the input manipulation. Hereinafter, the detailed descriptionof each operation is provided.

In operation S1200, the second client receives input manipulation to theobject, from the second user. That is, the second client obtains adetailed cell condition corresponding to the input manipulation by thesecond user. The second client obtains location information (i.e., the Xcoordinate and Y coordinate at the point at which user inputmanipulation is applied) and depth information (i.e., the appliedpressure data or the time length in which a manipulation object contactsa screen) on the screen through input manipulation of the second user.

When successive input manipulations are applied by the second user, thesecond client obtains location information and depth information at aunit time interval. For example, when moving in a specific axisdirection (e.g., X-axis direction or Y-axis direction), the secondclient obtains the changed location information and the changed depthinformation at a unit time interval. Moreover, for example, when inputmanipulation is received from the second user while the pressurestrength is changed in a diagonal direction, the second clientsequentially obtains the changed location information and the changeddepth information.

In operation S1400, the second client sequentially extracts and providesan image frame in the reaction-type image based on the inputmanipulation. As the location information and depth information of theinput manipulation is changed, the computer extracts the image framematched to the detailed cell corresponding to the changed locationinformation and the changed depth information to continuously providethe extracted image frame.

As illustrated in FIG. 6, when input manipulation to point A, point B,point C moves while the pressure strength or contact time length of theinput manipulation increases, the second client searches for a detailedcell corresponding to the applied pressure strength or the contact timelength in a plurality of detailed cells of the location (e.g., thelocation where the finger is contacted) where the input manipulation isapplied and extracts the image frame matched to the detailed cell.

In particular, at point A, as pressure is not applied and a manipulationobject is contacted, the second client selects the detailed cell oflevel ‘0’; at point B, the second client selects the detailed cell oflevel ‘0’ based on the pressure strength; and at point C, the secondclient selects the detailed cell of level ‘4’ based on the pressurestrength. The second client sequentially provides image frames matchedto each detailed cell selected while moving from point A to point C.

In an embodiment, the second client, which is a computer, may receiveand play each image frame from the service server. For example, thesecond client transmits, to the service server, the location (e.g., apixel location on the screen to which input manipulation of the seconduser is applied) on the screen and the pressure strength measured by thepressure sensor or the time length contacted on the screen; the serviceserver searches for a specific detailed cell in the cell combinationbased on the transmitted result and provides the second client with theimage frame matched to the found result. The second client extracts theimage frame received from the service server on the screen. When thedelay according to communication between a service server and a clientis short, an image frame corresponding to location information and depthinformation (i.e., pressure strength or contact time length) of theinput manipulation may be displayed after a short time after the inputmanipulation of the second user is entered into the second client, andthus an image may be implemented as if a user is responding directly tosecond user manipulation.

Moreover, in another embodiment, after the second client receives theentire cell combination of the reaction-type image from the serviceserver or the first client, the second client searches for specificdetailed cells in the cell combination, based on location informationand depth information (i.e., the applied pressure strength or the timelength) according to the input manipulation at each point in time, andextracts the image frame matched to the found result to display theextracted image frame on the screen.

As such, without the need to separately include the image to be playedfor each input manipulation motion by a user, the second client mayimplement various object motions corresponding to the input manipulationof the second user, by store image frames according to each inputmanipulation condition (i.e., location information and depthinformation) in a database.

Moreover, in another embodiment, the second client may differentlydetermine a detailed cell that extracts a frame to be playedsequentially depending on the method for applying the inputmanipulation.

Moreover, in another embodiment of the inventive concept, the methodfurther includes generating (S1300), by a second client, a finalplayback image frame by performing morphing based on a plurality ofimage frames corresponding to a specific point on a touch screen (i.e.,screen) when successive manipulations are entered into adjacent regions.For example, when the computer stores the obtained image frame, which isobtained while the computer manipulates the object in the first axisdirection and the second axis direction as specific pressure is appliedby the user, as a base image, the computer generates an image frame onwhich morphing is performed, with respect to regions other than thepoint where touch manipulation is entered at a specific point in time,based on a plurality of image frames. As such, a reaction-type imagechanged naturally during the user's manipulation may be generated.

In particular, as the image frame provided at the first time point, thelocation of a finger, and pressure strength are changed, the secondclient performs morphing based on the image frame provided at the secondtime point (a point in time when the specific time elapses from thefirst time point). That is, the second client performs morphing usingthe image frame of a detailed cell corresponding to the current inputmanipulation and one or more previously provided image frames. Forexample, the second client may generate a final playback image frame byapplying the average value of the image frame at the first time pointand the image frame at the second time point to the same point.

As such, although the image frame in the detailed cell matched to theinput manipulation is not matched to the motion performed by the seconduser, the final playback image frame matched to the motion throughcorrection may be generated. That is, the sense of reality may beprovided by providing an image suitable for the motion provided by thesecond user, various image frames may be generated in a state where onlythe basic image frame is matched to each detailed cell and storedwithout the need to store image frames for all the motions, therebysaving storage space.

Moreover, the second client may transform the image region within thepredetermined range from a point at which the input manipulation isentered by the second user. That is, because deformation occurs in thesurrounding region as pressure is applied to a specific point, thesecond client performs correction on the surrounding region together.

The second client may perform correction such that the set inputmanipulation is represented as being continuously changed from theapplied point to the end of the set peripheral region. For example, thecorrection is performed such that the color is changed sequentiallybased on the color value at a point at which the input manipulation isapplied and the color value of the end of the surrounding region. Atthis time, the second client may utilize the image frame of a defaultstate to which the input manipulation is not applied.

The second client may differently determine the peripheral region rangeto be corrected, depending on the pressure strength of the inputmanipulation. For example, when the object to which the inputmanipulation is applied is a cushion, because the region where thesurface deformation occurs becomes wider as the strong pressure isapplied, the second client determines the deformation range depending onthe pressure intensity to provide the sense of reality upon playing thereaction-type image.

Moreover, in another embodiment, the method further includes adjusting(S1500), by the second client, an image frame change by extracting arestoration variable in a reaction-type image. That is, the secondclient adjusts the speed at which the image frame is changed after theinput manipulation is applied, based on the restoration variableprovided while the reaction-type image is included together. As such, animage may be played as if an actual object is manipulated.

Moreover, in another embodiment, the method further includes adjusting(S1600), by the second client, the image frame change according to inputmanipulation to an object by calculating the time elapsed from thespecific time point at which the reaction-type image is executed. Assuch, when a state change occurs over time in real, the characteristicof the object (e.g., fruit, bread, cement, or the like) with differentdegrees of deformation according to pressure may be implemented with areal-world state change even in the reaction-type image; users may feelthe change of hardness of object as time goes on, through thereaction-type image. For example, the second client may change the imageframe with less object deformation to the last frame provided dependingon pressure or may adjust the speed of an image frame change, and thusit is realized that bread is hardened and puffiness of the bread isreduced over time in a reaction-type image and it is realized thatcement, clay, or the like is hardened over time in a reaction-typeimage.

The multidimensional reaction-type image generating device according toanother embodiment of the inventive concept includes a controller. Thecontroller performs a multidimensional reaction-type image generatingmethod according to embodiments of the inventive concept.

Moreover, another embodiment of a multidimensional reaction-type imagegenerating device includes an image capturing unit (i.e., camera) forcapturing a base image.

According to an embodiment of the inventive concept described above, amethod for generating or playing a multidimensional reaction-type imagemay be implemented as a program (or application) to be executed incombination with a hardware computer and may be stored in a medium.

The above-described program may include a code encoded by using acomputer language such as C, C++, JAVA, a machine language, or the like,which a processor (CPU) of the computer can read through the deviceinterface of the computer, such that the computer reads the program andperforms the methods implemented with the program. The code may includea functional codes associated with the function that defines functionsnecessary to perform the methods, and may include a control codeassociated with an execution procedure necessary for the processor ofthe computer to perform the functions in a predetermined procedure.Furthermore, the code may further include additional informationnecessary for the processor of the computer to perform the functions ora memory reference-related code associated with the location (address)of the internal or external memory of the computer, at which the medianeeds to be checked. Moreover, when the processor of the computer needsto communicate with any other remote computer or any other remote serverto perform the functions, the code may further include acommunication-related code associated with how to communicate with anyother remote computer or server using the communication module of thecomputer, what information or media should be transmitted or receivedduring communication, or the like.

The stored media may mean the media that does not store data for a shortperiod of time such as a register, a cache, a memory, or the like butsemi-permanently stores to be read by the device. Specifically, forexample, the stored media include, but are not limited to, ROM, RAM,CD-ROM, magnetic tape, floppy disk, optical data storage device, and thelike. That is, the program may be stored in various recording media onvarious servers that the computer can access, or various recording mediaon the computer of the user. In addition, the media may be distributedto a computer system connected to a network, and a computer-readablecode may be stored in a distribution manner.

Although embodiments of the inventive concept have been described hereinwith reference to accompanying drawings, it should be understood bythose skilled in the art that the inventive concept may be embodied inother specific forms without departing from the spirit or essentialfeatures thereof. Therefore, it should be understood that the aboveembodiments are not limiting, but illustrative.

The inventive concept has the following various effects.

First, the reaction-type image may be played as if the surface of anobject is changed in real when a user performs the touch manipulation,thereby providing a realistic playback image to the user.

Second, the viewer's interest in the image may increase and the deliverymay be maximized. As such, the publicity effect on a specific object inthe image may be improved.

Third, when generating an image in which a specific operation isrepeated or an image including an operation that reciprocates a specificsection, such as stroking an object, only the image moving in a specificfirst direction may be obtained, and then the image may be repeatedlyplayed (e.g., repeat forward direction and reverse direction playback)depending on a user's input manipulation. As such, the storage capacityof the image including the repeated operation may be reduced.

Fourth, a realistic final playback image frame corresponding to themanipulation provided on the screen by the user may be provided byperforming morphing based on a basic image frame stored in each detailedcell. Accordingly, because it is not necessary to store different imageframes for each motion type, the storage capacity may be reducedsignificantly.

While the inventive concept has been described with reference toexemplary embodiments, it will be apparent to those skilled in the artthat various changes and modifications may be made without departingfrom the spirit and scope of the inventive concept. Therefore, it shouldbe understood that the above embodiments are not limiting, butillustrative.

What is claimed is:
 1. A method for playing a multidimensionalreaction-type image, the method comprising: receiving, by a computer,input manipulation to an object from a user; extracting, by thecomputer, an image frame matched to a detailed cell corresponding tolocation information and depth information in a reaction-type image,depending on the location information and the depth information of theinput manipulation received at each playback time point, wherein thedepth information is information about pressure strength of the inputmanipulation applied to the reaction-type image or time length to whichthe input manipulation is applied, wherein the location information isinformation about a location of a two-dimensional space in which theinput manipulation is applied to the reaction-type image, and wherein aspecific image frame corresponding to the location information and thedepth information of the input manipulation is matched to the detailedcell, and the detailed cell constitutes a three-dimensional cellcombination in the reaction-type image; and when successive inputmanipulations are entered into an adjacent region, generating, by thecomputer, a final playback image frame by performing morphing based on aplurality of image frame combinations, the depth informationcorresponding to a specific point on a screen of each of which isdifferent, or a combination of an image frame provided at a previoustime point and an image frame within the detailed cell corresponding toa current input manipulation.
 2. The method of claim 1, furthercomprising: adjusting, by the computer, speed of image frame change byextracting a restoration variable from the reaction-type image.
 3. Themethod of claim 1, further comprising: adjusting, by the computer, animage frame change according to the input manipulation to the object bycalculating time elapsing from a specific time point at which thereaction-type image is executed.
 4. A non-transitory computer-readablerecording medium storing a reaction-type image generating program, andconfigured to be coupled to a computer being hardware, the programincluding instructions to execute the method of claim 1.